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Abstract. The demands of cutting-edge science are driving the need for larger and faster 
computing resources. With the rapidly growing scale of computing systems and the prospect 
of technologically disruptive architectures to meet these needs, scientists face the challenge 
of effectively using complex computational resources to advance scientific discovery. Multi- 
disciplinary collaborating networks of researchers with diverse scientific backgrounds are 
needed to address these complex challenges. The UNEDF SciDAC collaboration of nuclear 
theorists, applied mathematicians, and computer scientists is developing a comprehensive 
description of nuclei and their reactions that delivers maximum predictive power with quantified 
uncertainties. This paper describes UNEDF and identifies attributes that classify it as a 
successful computational collaboration. We illustrate significant milestones accomplished by 
UNEDF through integrative solutions using the most reliable theoretical approaches, most 
advanced algorithms, and leadership-class computational resources. 



1. Introduction 

One of the discovery frontiers in physics is to explain the nature of atomic nuclei. Apart from 
a plethora of basic science interests, this is also an essential component of energy, medical, and 
biological research, and national security. Nuclear physicists are working toward a fundamental 
and unified description of nuclei based on the underlying theory of the strong interactions, 
quantum chromodynamics, and to transform descriptive and highly phenomenological models 
into predictive capability; in particular, allowing reliable extrapolations into regions that are 
not accessible by experiments. Ultimately, this would allow for accurate predictions of nuclear 
reactions, with significant impact on the development of advanced fission reactors and fusion 
energy sources, and in industrial and medical innovations through the use of stable isotopes and 
radioisotopes. 

The UNEDF collaboration of nuclear theorists, applied mathematicians, and computer 
scientists (see Fig. IT]) is making significant strides toward realizing this goal through a 



comprehensive study of all nuclei built on the latest advances in nuclear theory and scientific 
computing. UNEDF, which stands for "Universal Nuclear Energy Density Functional," is 
a five-year SciDAC ("Scientific Discovery through Advanced Computing") project [TJ [2]. 

The SciDAC program (www.scidac.gov) has 
provided the opportunity for applied math- 
ematicians and computer scientists to work 
collaboratively with physicists to develop 
and interconnect the most accurate knowl- 
edge of the strong nuclear interaction, high- 
precision theoretical approaches, scalable al- 
gorithms, and high-performance computing 
tools and libraries to enable scientific dis- 
coveries using leadership-class computing re- 
sources. Working toward a predictive the- 
ory, the UNEDF project emphasizes the ver- 
ification of methods and codes, the estima- 
tion of uncertainties, and the assessment of 
results. An added and unexpected benefit 
of the UNEDF project has been the realiza- 
tion of new physics collaborations, identified 
through shared computational methods and 
needs. Here we present an overview of UN- 
EDF, some significant milestones achieved 
Figure 1. UNEDF involves over 50 researchers through the UNEDF collaborative effort and 
from 9 universities and 7 national laboratories. the outlook for the future; more details and 
Annually, it provides training to about 30 young refe rences can be found at the UNEDF web- 
researchers (postdocs and students). site [http://www.unedf.org 

2. UNEDF Overview 

There are approximately 3,000 known nuclei, most of them produced in the laboratory, and an 
estimated 6,000 nuclei that could in principle still be created in nuclear laboratories and in the 
Cosmos. Understanding the properties of these nuclei is crucial for a complete nuclear theory, 
for element formation, for properties of stars, and for present and future energy and defense 
applications. Figure [2] shows the nuclear landscape as a function of neutron and proton number. 
Overlaying the nuclear landscape are the regions applicable for the major theoretical approaches 
and computational techniques utilized in the UNEDF collaboration: ab initio, configuration 
interaction, and density functional theory. Methods are applicable to a particular mass of nuclei 
and constrained by the computational resources available. Furthermore, by investigating the 
intersections and overlaps of these regions, UNEDF members gain valuable input for establishing 
a robust theory with high-quality predictive power. 

2.1. Strategy Diagram 

The UNEDF collaboration involves a synthesis of different perspectives to face the challenge 
of understanding the low-energy nuclear many-body problem within the landscape of high- 
performance computing. Accomplishing the scientific goals requires development of integrative 
solutions that extend beyond the capabilities of a single domain while maintaining a clear, unified 
strategy. 

The UNEDF strategy diagram shown in Fig.[3]displays the cohesive view of the collaboration's 
strategy [2]. This diagram shows the major scientific focus areas and provides necessary 
granularity to the overlapping computational methods seen in Fig. [2] Through the strategy 





Figure 2. Theoretical methods and computational techniques to solve the nuclear many- 
body problem across the nuclear landscape. The thick dotted lines indicate domains of 
major theoretical approaches to the nuclear many-body problem. For the lightest nuclei, 
ab initio calculations based on the bare nucleon-nucleon and three- nucleon interactions, 
are possible (red). Medium-mass nuclei can be treated by configuration interaction (CI) 
techniques (interacting shell model (green)). For heavy nuclei, the density functional theory, 
based on self-consistent/mean- field theory (blue), is the tool of choice. The red vertical and 
horizontal lines show the magic numbers, reflecting regions where nuclei are expected to be 
more tightly bound and have longer half-lives. The anticipated path of the astrophysical 
r-process responsible for nucleosynthesis of heavy elements is also shown (purple line). 
Adapted from Ref. pQ. 

diagram, UNEDF conveys the interdependence of the various focus areas of the collaboration 
and identifies the challenges where multidomain expertise is necessary. It also provides a 
meaningful division of labor as well as a global perspective on the impact from individual efforts. 
Furthermore, the diagram in Fig. [3] helps to identify and foster unexpected cross-cutting physics 
research and shared computational challenges. Clearly outlining the efforts serves to maintain 
focus and heighten the collaborative spirit within the group. 

2.2. Measures of Success 

The success of the SciDAC UNEDF collaboration can be quantitatively measured by the 
scientific impact of the research performed. Figure [4] shows the number of publications 
resulting from UNEDF research over five years. Important to note are the high number 
of Physical Review Letters as the collaboration has matured; a Science highlight [5j 
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Figure 3. UNEDF strategy diagram identifying the interconnections between the four 
primary focus areas: ab initio, configuration interaction, nuclear density functional 
theory and extensions, and compound nuclear and direct reaction theory. Taken from 
|http: / /unedf .org[ 
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Figure 4. Publications produced yearly 
through the UNEDF collaboration. The solid 
line shows all publications and the dashed-line 
shows cross-domain authorship on publication. 

Number of publications for 2011 is incomplete. 

in the increased collaboration across domains, 
future research and address the challenges pose( 



in 2011; and cross-domain authorship in 
nuclear physics, physics, and computing 
publications. 

The UNEDF effort has also placed great 
importance on recruiting and retaining the 
next generation leaders in low-energy nuclear 
physics, applied mathematics, computer sci- 
ence, and high-performance computing. An- 
nually, it has provided training to approxi- 
mately 30 young researchers, including post- 
docs and graduate students. Through ex- 
perience in the UNEDF collaboration, re- 
searchers have received job placement in na- 
tional laboratories and faculty positions at 
universities, as well as received various fund- 
ing and awards. A list of publications, one- 
page highlights, and additional information 
on awards and appointments can be found at 
the UNEDF website. 

Qualitatively, UNEDF success can be seen 
These connections help lay the foundation for 
1 by emerging architectures. 



3. High-Performance Computing Resources 

Access to leadership-class computing resources and large compute time allocations are critical 
to the scientific investigations of many UNEDF members. Through the competitive INCITE 
("Innovative and Novel Computational Impact on Theory and Experiment") program, UNEDF 
members have been awarded large allocations on leadership-class computing resources at the 
Oak Ridge Leadership Computing Facility (OLCF) and the Argonne Leadership Computing 
Facility (ALCF). Computing resources include the following: 

Intrepid (ALCF), an IBM Blue Gene/P system with 40 racks containing 1024 nodes per rack 
and 850 MHz quad-core processors and 2 GB RAM per node. Intrepid currently provides 
users with 163,840 cores, roughly 82 TB of memory, 7.6 PB of disk space, and 88 GB/s of 
disk bandwidth. 

Jaguar (OLCF), a Cray XT with two partitions. The XT4 partition contains 7,832 compute 
nodes with quadcore AMD Opteron 1354 (Budapest) processors and 8 GB RAM per node, 
totaling 31,328 processing cores. The XT5 partition contains 18,688 compute nodes with 
dual hex-core AMD Opteron 2435 (Istanbul) processors and 16 GB RAM per node, totaling 
224,256 processing cores. Jaguar currently provides users with a peak performance of 2.332 
PF, 299 TB of system memory, 10 PB of disk space, and 240 GB/s of disk bandwidth. 
Note: The XT4 partition was the primary resource in 2008 and was retired in 2011. The 
XT5 partition was available in 2009 with dual quad-core AMD Opteron 2356 (Barcelona) 
processors and was upgraded to hex-core in 2010. 

Through the UNEDF collaboration, members have been able to continuously scale codes to 
efficiently utilize these ever-increasing resources. Figures |H and © show the INCITE allocat ions 
awarded to UNEDF collaborators and the CPU-hour utilization starting from 2008. These 
figures highlight the increasing demand for computing time in low-energy nuclear physics 
research. The combined utilization across Jaguar and Intrepid in 2008 was nearly 20 million 
CPU-hours and has increased more than threefold in 2011. For the 2012 calendar year, UNEDF 



members were granted the sixth largest allocation of the 60 INCITE projects awarded. These 
statistics show that low-energy nuclear physics research is dependent on high-performance 
computing and equally that low-energy nuclear physics is a scientific driver in HPC. 
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Figure 5. INCITE allocation and utilization 
of the Jaguar supercomputer at the OLCF 
(CY 2008-2013) 
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Figure 6. INCITE allocation and utilization 
of the Intrepid supercomputer at the ALCF 
(CY 2008-2013) 



Leadership-class utilization is an important metric for assessing the need for and usability 
of capability computing resources. Capability computing is defined as "using the maximum 
computing power to solve a large problem in the shortest amount of time," thus solving "a 
problem of a size or complexity that no other computer can" [4J. This is in contrast with 
capacity computing, which uses "efficient cost-effective computing power to solve somewhat 
large problems or many small problems." At the OLCF, utilization of Jaguar is binned into 
three job-size categories: usage of less than 20%, between 20 and 60%, and greater than 60% of 
the computing resource for a single job. The typical scale of a code appropriate for a leadership- 
class system is utilization of greater than 20% of the resource for a single calculation. Job sizes 
less than 20% of the resource can typically fit onto smaller capacity systems. 



Table 1. Utilization of Jaguar Supercomputer 



Year 


Allocation 


Usage (CPU-Hours) 


Leadership 


2011 


28,000,000 


50,076,810 


66% 


2010 


25,000,000 


28,465,982 


61% 


2009 


15,000,000 


23,859,172 


78% 


2008 


7,500,000 


8,432,335 


65% 



Table [T] and Fig. [7] show utilization by UNEDF projects of the Jaguar supercomputer 
binned by job size. Table [I] shows that UNEDF projects consistently use over 60% of 
their allocation for leadership-size jobs. It is important to note that in 2008, when 
Jaguar was an XT4, leadership-class jobs used more than 6,266 cores, in 2009 for the 
Jaguar XT5 quad-core jobs used more than 29,901 cores, and since 2010 for the Jaguar 
XT5 hex-core jobs used more than 44,852 cores. Figure [7] provides additional granularity 
to show that UNEDF projects require 60% of the resource for a single computational 
run for nearly 25% of their usage. This shows the success of UNEDF collaborations 
to continually meet the changing architecture and growing size of computing systems. 



100% 



so% 



m 40% 



20% 




60% 



Additional computing time 
was provided in 2009 through 
the OLCF Early Science pe- 
riod prior to transitioning the 
general user population onto 
the Jaguar XT5 quad-core. 
At that time, the XT5 parti- 
tion had 18,688 compute nodes 
with dual quadcore AMD 
Opteron 2356 (Barcelona) pro- 
cessors, totaling 149,504 pro- 
cessing cores. The XT5 par- 
tition became available to the 
larger user community in July 
2009; for the first half of 2009, 

during its transition-to-operations period, it was open only to select Early Science users. UN- 
EDF members were awarded 30 million CPU-hours for an Early Science project on the XT5. 
Figure [8] shows the utilization by job size of over 350 million CPU-hours over six months by 26 
projects. The low-energy nuclear physics project labeled NPH009 shows that over 95% of its 
utilization was at leadership class [3J . 
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Figure 7. Utilization of Jaguar. 
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Figure 8. Utilization of Jaguar XT5 by job size for Early Science projects [3J. 



Close collaboration between nuclear physicists, applied mathematicians and computer 
scientists enable UNEDF research to effectively utilize high-performance computing resources, 
leading to the science highlights presented here. 



4. High-Performance Computing Enhances 
Calculations 
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Figure 9. MFDn simulation shows three- 
body forces are necessary to explain the 
anomalously long half-life of isotope carbon- 
14 used in carbon dating jS]. 
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Figure 10. NUCCOR calculations for 
medium-mass nuclei calcium-48 shows three- 
nucleon forces account for the missing binding 
energy [8]. 



Ab initio, or from first principles, nuclear structure calculations have made major 
advances under UNEDF toward effectively utilizing high-performance computing resources and 
transforming to meet the challenges posed by emerging architectures. Ab initio techniques 
provide a fine-grained method for studying nuclei and the nuclear interaction, but they 
often come with a high computational cost. They are necessary to the UNEDF effort by 
providing "control data" to constrain more general functionals and test candidate energy density 
functionals even for systems not experimentally accessible. UNEDF collaborators continue 
to scale ab initio nuclear structure simulations and perform the largest and most accurate 
calculations currently possible on both Jaguar (ORNL) and Intrepid (ANL). 

Examples of UNEDF-directed advances include development of the Asynchronous Dynamic 
Load Balancing (ADLB) software library by using Green's function Monte Carlo (GFMC) 
calculations as a testbed. The ADLB library has enabled GFMC to run efficiently on over 100,000 
cores on Intrepid [6J. Utilizing Jaguar, UNEDF applications Nuclear Coupled- Cluster - Oak 
Ridge (NUCCOR) and Many Fermion Dynamics-nuclear (MFDn) have undergone considerable 
code and algorithm development. Improvements include implementation of a hybrid MPI and 
OpenMP approach for efficient memory management, memory- aware algorithms, and integration 
of libraries and tools to enable further scaling for higher-precision calculations. 

Recent scientific breakthroughs include using NUCCOR to calculate medium-mass nuclei 



from the ground up starting from nucleon degrees of freedom, such as Ca shown in Figure 10 



[8J. Results show chiral nucleon- nucleon interactions perform remarkably well; the 400 keV per 
nucleon missing binding energy in 48 Ca can be attributed to chiral three- nucleon forces missing 
in calculations [8j. Another recent highlight explains the useful but anomalously long lifetime 
of 14 C by identifying the critical role of the three-nucleon force in its beta decay seen in Fig. [9] 
[9j. These calculations involve diagonalization of a Hamiltonian matrix of dimension 2 billion, 
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Figure 11. The optimization algorithm POUNDerS yields dramatic computational savings 
over alternative optimization methods (left). The resulting parameterization UNEDFO 
obtained by using the POUNDerS algorithm provides a baseline of nuclear ground-state 
properties to compare with future functionals (right) [10J. 



using 214,668 cores on the Jaguar supercomputer at ORNL under the Early Science projects 
allocation of 30 million CPU-hours. 



5. Advanced Algorithms and Tools Define New Generation Energy Density 
Functionals 

The UNEDF project has devoted considerable effort to develop and improve the algorithmic and 
computational infrastructure needed to optimize candidate energy density functionals (EDF). 
These developments have resulted in new optimization tools that are broadly available to other 
science domains. An example is the optimization algorithm POUNDerS, which not only provides 
a computational savings over other methods, as shown in Fig. 11, but greatly improves the time 
to solution to test candidate EDFs. With the derivative-free POUNDerS algorithm, the resulting 
parameterization of existing data yielded UNEDFO, which sets a solid baseline of nuclear ground- 
state properties to compare with future functionals shown in Figure [IT] [10J. Continuing work 
involves utilizing this approach to study new hybrid functionals with microscopic input from 
chiral effective field theory [TT] . 

These new tools enable for the first time, a consistent method for uncertainty quantification 
and correlation analysis to estimate errors and significance as a first step toward a formal process 
for future verification and validation. Included in the UNEDF project are the development 
and application of statistical tools, particularly important for directing future experiments by 
providing analysis of the significance of new experimental data. For example, the sensitivity 
of two optimized functionals to particular data is shown in Fig. [12] [TO] . Such capabilities have 
not been previously available in the low-energy nuclear theory community but are increasingly 
important as new theories and computational tools are applied to new nuclear systems and to 
conditions inaccessible to experiment. 



6. Massively Parallel Algorithms Open Cold Atoms as a Testing Ground 

UNEDF theorists have made important contributions to the study of strongly coupled superfluid 
systems such as ultracold Fermi atoms, which show many similarities to the cold nuclear matter 
found in the crust of neutron stars. Cold atoms make excellent laboratories for testing and 
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Figure 13. ASLDA simulation with strongly 
interacting spin-imbalanced atomic gases in 
extremely elongated traps [7] . 




Figure 14. TDSLDA simulation shows the 
ball-and-rod excitation of a unitary fermi gas 
with vortex formation [5] . 



improving the computational methods to be 
used for nuclei. Cold-atom systems also allow 
for predictions of superfluid DFT that are 
testable against experiment. 

UNEDF developments using cold atoms 
as a testing ground include adding new 
algorithms to existing applications, such 
as adapting the antisymmetric superfluid 
local density approximation (ASLDA) to an 
existing massively parallel nuclear DFT code 
with strongly interacting spin-imbalanced 
atomic gases in extremely elongated traps, 

® 



seen 



in Fig. 13 



Another major 
UNEDF development is implementation of 
the time-dependent superfluid local density 
approximation (TDSLDA) on a 3D spatial 
lattice [5]. Unlike previous methods, 
the UNEDF implementation eliminates the 
need for matrix operations, allowing it to 
accommodate a basis set that is 2-3 orders 
of magnitude larger than other approaches. 
Calculations [5j were performed by using 
97% of Jaguar to simulate the unitary gas 
(e.g., vortex formation) and a heavy nucleus 
under the action of various external fields, 
seen in Fig. 
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While still exploratory, 
these first-time simulations of this kind for 
fermion superfluids serve as proof of principle 
for an eventual treatment of neutron-induced 
fission. 




Figure 12. Statistical tools are used to deliver uncertainty quantification and error analysis 
for theoretical studies as well as to assess new experimental data [10J. 
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7. HPC Empowers New Era for Nuclear Reaction Theory 

One of the principal aims of the UNEDF project is to calculate nucleon- nucleus reactions crucial 
for 

astrophysics, nuclear energy radiobiology, 
and national security for which extensions 
of standard phenomenology is insufficient. 
Under the UNEDF effort, neutron reactions 
on heavier nuclei are being modeled by 
using DFT results to predict not just 
bound states but also scattering states 
for nucleons. As shown in Fig. 15, the 
calculated reaction cross-sections agree well 
with experimental data. For the first time, 
a complete microscopic calculation using 
basic interactions between nucleons can be 
used to predict reaction observables with 
low-incident energy [T2J - This technology 
provides the basis for future calculations 
of unstable species outside the range of 
experiment. 

Another important capability for reac- 
tions is the calculation of level densities, 
which provides insight to the interactions in- 
side the system. A new proton-neutron al- 
gorithm for the parallel JMoments code was 
recently designed and implemented, which 
scales to tens of thousands of cores and 
greatly increases the code's overall perfor- 
mance. This development opens the door to 
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Figure 15. New methods for calculating re- 
action cross-sections (black) show good agree- 
ment with experimental data. Total reaction 
cross section as a function of the incident en- 
ergy for the reaction p + 90Zr using the Gogny 
D1S force. The results are shown for couplings 
to the inelastic RPA states (red line), and to the 
inelastic and transfer channels with nonorthog- 
onality corrections (black line) [T2] . 



calculating accurate, nuclear level densities and reaction rates for a large class of nuclei [13j. 



8. Outlook 

The UNEDF collaboration has provided fertile ground for new and continuing growth between 
applied mathematics, computer science, and nuclear physics. Over the past five years, the 
collaboration has established cross-disciplinary working relationships to facilitate future efforts 
and has matured to adequately address new challenges in verification and validation, workflow, 
visualization, and new programming models with changing architectures. Reaching next- 
generation science objectives requires computational resources several orders of magnitude 
beyond what is currently available. Adapting to these changes will take conscious planning and 
purposeful action. The members of the collaboration are well positioned to meet these disruptive 
changes through the close working relationship established through UNEDF. UNEDF, and 
similar future collaborations, will continue to develop key computational codes and algorithms 
for reaching the goal of solving the nuclear quantum many-body problem, thus paving the road 
to the comprehensive model of the atomic nucleus. 
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